{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# The importance of constraints\n",
    "\n",
    "Constraints determine which potential adversarial examples are valid inputs to the model. When determining the efficacy of an attack, constraints are everything. After all, an attack that looks very powerful may just be generating nonsense. Or, perhaps more nefariously, an attack may generate a real-looking example that changes the original label of the input. That's why you should always clearly define the *constraints* your adversarial examples must meet. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/QData/TextAttack/blob/master/docs/2notebook/2_Constraints.ipynb)\n",
    "\n",
    "[![View Source on GitHub](https://img.shields.io/badge/github-view%20source-black.svg)](https://github.com/QData/TextAttack/blob/master/docs/2notebook/2_Constraints.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Classes of constraints\n",
    "\n",
    "TextAttack evaluates constraints using methods from three groups:\n",
    "\n",
    "- **Overlap constraints** determine if a perturbation is valid based on character-level analysis. For example, some attacks are constrained by edit distance: a perturbation is only valid if it perturbs some small number of characters (or fewer).\n",
    "\n",
    "- **Grammaticality constraints** filter inputs based on syntactical information. For example, an attack may require that adversarial perturbations do not introduce grammatical errors.\n",
    "\n",
    "- **Semantic constraints** try to ensure that the perturbation is semantically similar to the original input. For example, we may design a constraint that uses a sentence encoder to encode the original and perturbed inputs, and enforce that the sentence encodings be within some fixed distance of one another. (This is what happens in subclasses of `textattack.constraints.semantics.sentence_encoders`.)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### A new constraint\n",
    "\n",
    "To add our own constraint, we need to create a subclass of `textattack.constraints.Constraint`. We can implement one of two functions, either `_check_constraint` or `_check_constraint_many`:\n",
    "\n",
    "- `_check_constraint` determines whether candidate `TokenizedText` `transformed_text`, transformed from `current_text`, fulfills a desired constraint. It returns either `True` or `False`.\n",
    "- `_check_constraint_many` determines whether each of a list of candidates `transformed_texts` fulfill the constraint relative to `current_text`. This is here in case your constraint can be vectorized. If not, just implement `_check_constraint`, and `_check_constraint` will be executed for each `(transformed_text, current_text)` pair."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### A custom constraint\n",
    "\n",
    "\n",
    "For fun, we're going to see what happens when we constrain an attack to only allow perturbations that substitute out a named entity for another. In linguistics, a **named entity** is a proper noun, the name of a person, organization, location, product, etc. Named Entity Recognition is a popular NLP task (and one that state-of-the-art models can perform quite well). \n",
    "\n",
    "\n",
    "### NLTK and Named Entity Recognition\n",
    "\n",
    "**NLTK**, the Natural Language Toolkit, is a Python package that helps developers write programs that process natural language. NLTK comes with predefined algorithms for lots of linguistic tasks– including Named Entity Recognition.\n",
    "\n",
    "First, we're going to write a constraint class. In the `_check_constraints` method, we're going to use NLTK to find the named entities in both `current_text` and `transformed_text`. We will only return `True` (that is, our constraint is met) if `transformed_text` has substituted one named entity in `current_text` for another.\n",
    "\n",
    "Let's import NLTK and download the required modules:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package punkt to /u/edl9cy/nltk_data...\n",
      "[nltk_data]   Package punkt is already up-to-date!\n",
      "[nltk_data] Downloading package maxent_ne_chunker to\n",
      "[nltk_data]     /u/edl9cy/nltk_data...\n",
      "[nltk_data]   Package maxent_ne_chunker is already up-to-date!\n",
      "[nltk_data] Downloading package words to /u/edl9cy/nltk_data...\n",
      "[nltk_data]   Package words is already up-to-date!\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import nltk\n",
    "nltk.download('punkt') # The NLTK tokenizer\n",
    "nltk.download('maxent_ne_chunker') # NLTK named-entity chunker\n",
    "nltk.download('words') # NLTK list of words"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### NLTK NER Example\n",
    "\n",
    "Here's an example of using NLTK to find the named entities in a sentence:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(S\n",
      "  In/IN\n",
      "  2017/CD\n",
      "  ,/,\n",
      "  star/NN\n",
      "  quarterback/NN\n",
      "  (PERSON Tom/NNP Brady/NNP)\n",
      "  led/VBD\n",
      "  the/DT\n",
      "  (ORGANIZATION Patriots/NNP)\n",
      "  to/TO\n",
      "  the/DT\n",
      "  (ORGANIZATION Super/NNP Bowl/NNP)\n",
      "  ,/,\n",
      "  but/CC\n",
      "  lost/VBD\n",
      "  to/TO\n",
      "  the/DT\n",
      "  (ORGANIZATION Philadelphia/NNP Eagles/NNP)\n",
      "  ./.)\n"
     ]
    }
   ],
   "source": [
    "sentence = ('In 2017, star quarterback Tom Brady led the Patriots to the Super Bowl, '\n",
    "           'but lost to the Philadelphia Eagles.')\n",
    "\n",
    "# 1. Tokenize using the NLTK tokenizer.\n",
    "tokens = nltk.word_tokenize(sentence)\n",
    "\n",
    "# 2. Tag parts of speech using the NLTK part-of-speech tagger.\n",
    "tagged = nltk.pos_tag(tokens)\n",
    "\n",
    "# 3. Extract entities from tagged sentence.\n",
    "entities = nltk.chunk.ne_chunk(tagged)\n",
    "print(entities)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It looks like `nltk.chunk.ne_chunk` gives us an `nltk.tree.Tree` object where named entities are also `nltk.tree.Tree` objects within that tree. We can take this a step further and grab the named entities from the tree of entities:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[Tree('PERSON', [('Tom', 'NNP'), ('Brady', 'NNP')]), Tree('ORGANIZATION', [('Patriots', 'NNP')]), Tree('ORGANIZATION', [('Super', 'NNP'), ('Bowl', 'NNP')]), Tree('ORGANIZATION', [('Philadelphia', 'NNP'), ('Eagles', 'NNP')])]\n"
     ]
    }
   ],
   "source": [
    "# 4. Filter entities to just named entities.\n",
    "named_entities = [entity for entity in entities if isinstance(entity, nltk.tree.Tree)]\n",
    "print(named_entities)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Caching with `@functools.lru_cache`\n",
    "\n",
    "A little-known feature of Python 3 is `functools.lru_cache`, a decorator that allows users to easily cache the results of a function in an LRU cache. We're going to be using the NLTK library quite a bit to tokenize, parse, and detect named entities in sentences. These sentences might repeat themselves. As such, we'll use this decorator to cache named entities so that we don't have to perform this expensive computation multiple times."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Putting it all together: getting a list of Named Entity Labels from a sentence\n",
    "\n",
    "Now that we know how to tokenize, parse, and detect named entities using NLTK, let's put it all together into a single helper function. Later, when we implement our constraint, we can query this function to easily get the entity labels from a sentence. We can even use `@functools.lru_cache` to try and speed this process up."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "import functools\n",
    "\n",
    "@functools.lru_cache(maxsize=2**14)\n",
    "def get_entities(sentence):\n",
    "    tokens = nltk.word_tokenize(sentence)\n",
    "    tagged = nltk.pos_tag(tokens)\n",
    "    # Setting `binary=True` makes NLTK return all of the named\n",
    "    # entities tagged as NNP instead of detailed tags like\n",
    "    #'Organization', 'Geo-Political Entity', etc.\n",
    "    entities = nltk.chunk.ne_chunk(tagged, binary=True)\n",
    "    return entities.leaves()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And let's test our function to make sure it works:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('Jack', 'NNP'),\n",
       " ('Black', 'NNP'),\n",
       " ('starred', 'VBD'),\n",
       " ('in', 'IN'),\n",
       " ('the', 'DT'),\n",
       " ('2003', 'CD'),\n",
       " ('film', 'NN'),\n",
       " ('classic', 'JJ'),\n",
       " ('``', '``'),\n",
       " ('School', 'NNP'),\n",
       " ('of', 'IN'),\n",
       " ('Rock', 'NNP'),\n",
       " (\"''\", \"''\"),\n",
       " ('.', '.')]"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sentence = 'Jack Black starred in the 2003 film classic \"School of Rock\".'\n",
    "get_entities(sentence)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We flattened the tree of entities, so the return format is a list of `(word, entity type)` tuples. For non-entities, the `entity_type` is just the part of speech of the word. `'NNP'` is the indicator of a named entity (a proper noun, according to NLTK). Looks like we identified three named entities here: 'Jack' and 'Black', 'School', and 'Rock'. as a 'GPE'. (Seems that the labeler thinks Rock is the name of a place, a city or something.) Whatever technique NLTK uses for named entity recognition may be a bit rough, but it did a pretty decent job here!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Creating our NamedEntityConstraint\n",
    "\n",
    "Now that we know how to detect named entities using NLTK, let's create our custom constraint."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "from textattack.constraints import Constraint\n",
    "\n",
    "class NamedEntityConstraint(Constraint):\n",
    "    \"\"\" A constraint that ensures `transformed_text` only substitutes named entities from `current_text` with other named entities.\n",
    "    \"\"\"\n",
    "    def _check_constraint(self, transformed_text, current_text):\n",
    "        transformed_entities = get_entities(transformed_text.text)\n",
    "        current_entities = get_entities(current_text.text)\n",
    "        # If there aren't named entities, let's return False (the attack\n",
    "        # will eventually fail).\n",
    "        if len(current_entities) == 0:\n",
    "            return False\n",
    "        if len(current_entities) != len(transformed_entities):\n",
    "            # If the two sentences have a different number of entities, then \n",
    "            # they definitely don't have the same labels. In this case, the \n",
    "            # constraint is violated, and we return False.\n",
    "            return False\n",
    "        else:\n",
    "            # Here we compare all of the words, in order, to make sure that they match.\n",
    "            # If we find two words that don't match, this means a word was swapped \n",
    "            # between `current_text` and `transformed_text`. That word must be a named entity to fulfill our\n",
    "            # constraint.\n",
    "            current_word_label = None\n",
    "            transformed_word_label = None\n",
    "            for (word_1, label_1), (word_2, label_2) in zip(current_entities, transformed_entities):\n",
    "                if word_1 != word_2:\n",
    "                    # Finally, make sure that words swapped between `x` and `x_adv` are named entities. If \n",
    "                    # they're not, then we also return False.\n",
    "                    if (label_1 not in ['NNP', 'NE']) or (label_2 not in ['NNP', 'NE']):\n",
    "                        return False            \n",
    "            # If we get here, all of the labels match up. Return True!\n",
    "            return True\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "### Testing our constraint\n",
    "\n",
    "We need to create an attack and a dataset to test our constraint on. We went over all of this in the transformations tutorial, so let's gloss over this part for now."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "83cfddb4681b46d38a7155ca4ebdbbda",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, description='Downloading', max=736.0, style=ProgressStyle(description_…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "934709ba69334a34b63e63555b50b72a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, description='Downloading', max=46747112.0, style=ProgressStyle(descrip…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "00a1ee97f37e4dd79357731dfe21ed2d",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, description='Downloading', max=760289.0, style=ProgressStyle(descripti…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d49c0092188640028cc5b36648794867",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, description='Downloading', max=156.0, style=ProgressStyle(description_…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "62d9b14ed049497f9f9ac56372681421",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, description='Downloading', max=25.0, style=ProgressStyle(description_w…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[34;1mtextattack\u001b[0m: Goal function <class 'textattack.goal_functions.classification.untargeted_classification.UntargetedClassification'> compatible with model AlbertForSequenceClassification.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "4a63bb6d448e4867965a00293f0c72ce",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, description='Downloading', max=5787.0, style=ProgressStyle(description…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "2b34e451729d47b1b407a88697439f44",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, description='Downloading', max=3419.0, style=ProgressStyle(description…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Downloading and preparing dataset yelp_polarity/plain_text (download: 158.67 MiB, generated: 421.28 MiB, total: 579.95 MiB) to /u/edl9cy/.cache/huggingface/datasets/yelp_polarity/plain_text/1.0.0...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e586d9ff3e784be58e0cd6e6389444f7",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, description='Downloading', max=166373201.0, style=ProgressStyle(descri…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=1.0, bar_style='info', max=1.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=1.0, bar_style='info', max=1.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[34;1mtextattack\u001b[0m: Loading \u001b[94mnlp\u001b[0m dataset \u001b[94myelp_polarity\u001b[0m, split \u001b[94mtest\u001b[0m.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dataset yelp_polarity downloaded and prepared to /u/edl9cy/.cache/huggingface/datasets/yelp_polarity/plain_text/1.0.0. Subsequent calls will reuse this data.\n"
     ]
    }
   ],
   "source": [
    "# Import the model\n",
    "import transformers\n",
    "from textattack.models.tokenizers import AutoTokenizer\n",
    "from textattack.models.wrappers import HuggingFaceModelWrapper\n",
    "\n",
    "model = transformers.AutoModelForSequenceClassification.from_pretrained(\"textattack/albert-base-v2-yelp-polarity\")\n",
    "tokenizer = AutoTokenizer(\"textattack/albert-base-v2-yelp-polarity\")\n",
    "\n",
    "model_wrapper = HuggingFaceModelWrapper(model, tokenizer)\n",
    "\n",
    "# Create the goal function using the model\n",
    "from textattack.goal_functions import UntargetedClassification\n",
    "goal_function = UntargetedClassification(model_wrapper)\n",
    "\n",
    "# Import the dataset\n",
    "from textattack.datasets import HuggingFaceDataset\n",
    "dataset = HuggingFaceDataset(\"yelp_polarity\", None, \"test\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Attack(\n",
      "  (search_method): GreedySearch\n",
      "  (goal_function):  UntargetedClassification\n",
      "  (transformation):  WordSwapEmbedding(\n",
      "    (max_candidates):  15\n",
      "    (embedding_type):  paragramcf\n",
      "  )\n",
      "  (constraints): \n",
      "    (0): NamedEntityConstraint(\n",
      "        (compare_against_original):  False\n",
      "      )\n",
      "    (1): RepeatModification\n",
      "    (2): StopwordModification\n",
      "  (is_black_box):  True\n",
      ")\n"
     ]
    }
   ],
   "source": [
    "from textattack.transformations import WordSwapEmbedding\n",
    "from textattack.search_methods import GreedySearch\n",
    "from textattack.shared import Attack\n",
    "from textattack.constraints.pre_transformation import RepeatModification, StopwordModification\n",
    "\n",
    "# We're going to the `WordSwapEmbedding` transformation. Using the default settings, this\n",
    "# will try substituting words with their neighbors in the counter-fitted embedding space. \n",
    "transformation = WordSwapEmbedding(max_candidates=15) \n",
    "\n",
    "# We'll use the greedy search method again\n",
    "search_method = GreedySearch()\n",
    "\n",
    "# Our constraints will be the same as Tutorial 1, plus the named entity constraint\n",
    "constraints = [RepeatModification(),\n",
    "               StopwordModification(),\n",
    "               NamedEntityConstraint(False)]\n",
    "\n",
    "# Now, let's make the attack using these parameters. \n",
    "attack = Attack(goal_function, constraints, transformation, search_method)\n",
    "\n",
    "print(attack)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let's use our attack. We're going to attack samples until we achieve 5 successes. (There's a lot to check here, and since we're using a greedy search over all potential word swap positions, each sample will take a few minutes. This will take a few hours to run on a single core.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1 of 5 successes complete.\n",
      "2 of 5 successes complete.\n",
      "3 of 5 successes complete.\n",
      "4 of 5 successes complete.\n",
      "5 of 5 successes complete.\n"
     ]
    }
   ],
   "source": [
    "from textattack.loggers import CSVLogger # tracks a dataframe for us.\n",
    "from textattack.attack_results import SuccessfulAttackResult\n",
    "\n",
    "results_iterable = attack.attack_dataset(dataset)\n",
    "logger = CSVLogger(color_method='html')\n",
    "\n",
    "num_successes = 0\n",
    "while num_successes < 5:\n",
    "    result = next(results_iterable)\n",
    "    if isinstance(result, SuccessfulAttackResult):\n",
    "        logger.log_attack_result(result)\n",
    "        num_successes += 1\n",
    "        print(f'{num_successes} of 5 successes complete.')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's visualize our 5 successes in color:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>original_text</th>\n",
       "      <th>perturbed_text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Picture Billy Joel's \\\"\"Piano Man\\\"\" <font color = green>DOUBLED</font> mixed with beer, a rowdy crowd, and comedy - Welcome to Sing Sing!  A unique musical experience found in <font color = green>Homestead</font>.\\<font color = green>n</font>\\nIf you're looking to grab a bite to eat or a beer, come on in!  Serving food and brews from Rock <font color = green>Bottom</font> <font color = green>Brewery</font>, Sing Sing keeps your tummy full while you listen to two (or more) amazingly talented pianists take your musical requests.  They'll play anything you'd like, for tips of course.  Wanting to hear Britney Spears?  Toto?  Duran Duran?  <font color = green>Yep</font>, they play that... new or old.\\n\\nThe crowd makes the show, so make sure you come ready for a good time.  If the crowd is dead, it's harder for the Guys to get a reaction.  If you're wanting to have some fun, it can be a GREAT time!  It's the perfect place for Birthday parties - especially if you want to embarrass a friend.  The guys will bring them up to the pianos and perform a little ditty.  For being a good sport, you get the coveted <font color = green>Sing</font> <font color = green>Sing</font> bumper sticker.  Now who wouldn't want that?\\n\\nDueling Pianos and brews... time to Shut Up & Sing Sing!</td>\n",
       "      <td>Picture Billy Joel's \\\"\"Piano Man\\\"\" <font color = red>DOPPELGANGER</font> mixed with beer, a rowdy crowd, and comedy - Welcome to Sing Sing!  A unique musical experience found in <font color = red>Fairview</font>.\\<font color = red>nope</font>\\nIf you're looking to grab a bite to eat or a beer, come on in!  Serving food and brews from Rock <font color = red>Inferior</font> <font color = red>Stout</font>, Sing Sing keeps your tummy full while you listen to two (or more) amazingly talented pianists take your musical requests.  They'll play anything you'd like, for tips of course.  Wanting to hear Britney Spears?  Toto?  Duran Duran?  <font color = red>Alrighty</font>, they play that... new or old.\\n\\nThe crowd makes the show, so make sure you come ready for a good time.  If the crowd is dead, it's harder for the Guys to get a reaction.  If you're wanting to have some fun, it can be a GREAT time!  It's the perfect place for Birthday parties - especially if you want to embarrass a friend.  The guys will bring them up to the pianos and perform a little ditty.  For being a good sport, you get the coveted <font color = red>Blackmailing</font> <font color = red>Blackmailing</font> bumper sticker.  Now who wouldn't want that?\\n\\nDueling Pianos and brews... time to Shut Up & Sing Sing!</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>When I think BBB... I think the days of simply bringing your bike in for a quick and relatively inexpensive tune-up and a few fixes are long gone.    \\n\\nThis review is more for the repair end of BBB. In their defense BBB does appear to carry some amazing brands of bike (ie Colnago) that you just don't find anywhere else in Pittsburgh. \\n\\nAt BBB I was charged $250 for a tune up and a few other things. Granted this included installing a sew up tire (which I can understand would cost approx $50), Swapping out a left side (big ring) shifter on my down tube (this should have cost approx. $20 at most) and installing new bar tape (cost of tape $20 and $20 to install?).. SO WHAT\\\"\"S WITH $140 FOR A <font color = red>TUNE</font> <font color = red>UP</font>? <font color = red>Well</font> the story goes like this:\\n\\nI bring the bike into BBB prior to the nice weather hitting <font color = red>Pittsburgh</font> in hopes of trying what people have said is a great bike shop and getting my OCLV TREK 5900 ready for the season. Turns out I don't hear from these guys. A week goes by ...two weeks...I think that's ok I have two or three other bike I can turn to for a ride. Then I wind up going out of town for a week thinking for sure I'll get a call from them re: my bike is ready to roll...but no dice. So I call. Turns out a screw snapped when the mechanic was re-installing the down tube shifter and it had to be tapped out (is that my fault?). He says \\\"\"Should be ready in a few days\\\"\". So I come in a few days later to this mammoth bill. I ask if I am paying for the labor of taping out the screw? I don't think I ever got a straight answer? I look at the bill and can't see a good breakdown of the charges. Normally I would \\\"\"duke it over\\\"\" a bill like this but I figured...I had somewhere I to be 10 minutes ago and at least I finally have my bike. I would expect that for that money my bike could have been stripped down to the frame and totally gone over (overhauled). But it wasn't.  Well BBB I'll give you a star because the mechanic did do a good job in that my cycle shifts well and the tape job on the bars looks great (nice wrap). Plus I'll toss in a star for your outstanding selection of high end cycles. Maybe I would have rated BBB higher if I was in the market for a purchase instead of a simple repair?</td>\n",
       "      <td>When I think BBB... I think the days of simply bringing your bike in for a quick and relatively inexpensive tune-up and a few fixes are long gone.    \\n\\nThis review is more for the repair end of BBB. In their defense BBB does appear to carry some amazing brands of bike (ie Colnago) that you just don't find anywhere else in Pittsburgh. \\n\\nAt BBB I was charged $250 for a tune up and a few other things. Granted this included installing a sew up tire (which I can understand would cost approx $50), Swapping out a left side (big ring) shifter on my down tube (this should have cost approx. $20 at most) and installing new bar tape (cost of tape $20 and $20 to install?).. SO WHAT\\\"\"S WITH $140 FOR A <font color = green>MELODIES</font> <font color = green>ARRIBA</font>? <font color = green>Too</font> the story goes like this:\\n\\nI bring the bike into BBB prior to the nice weather hitting <font color = green>Philly</font> in hopes of trying what people have said is a great bike shop and getting my OCLV TREK 5900 ready for the season. Turns out I don't hear from these guys. A week goes by ...two weeks...I think that's ok I have two or three other bike I can turn to for a ride. Then I wind up going out of town for a week thinking for sure I'll get a call from them re: my bike is ready to roll...but no dice. So I call. Turns out a screw snapped when the mechanic was re-installing the down tube shifter and it had to be tapped out (is that my fault?). He says \\\"\"Should be ready in a few days\\\"\". So I come in a few days later to this mammoth bill. I ask if I am paying for the labor of taping out the screw? I don't think I ever got a straight answer? I look at the bill and can't see a good breakdown of the charges. Normally I would \\\"\"duke it over\\\"\" a bill like this but I figured...I had somewhere I to be 10 minutes ago and at least I finally have my bike. I would expect that for that money my bike could have been stripped down to the frame and totally gone over (overhauled). But it wasn't.  Well BBB I'll give you a star because the mechanic did do a good job in that my cycle shifts well and the tape job on the bars looks great (nice wrap). Plus I'll toss in a star for your outstanding selection of high end cycles. Maybe I would have rated BBB higher if I was in the market for a purchase instead of a simple repair?</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>The first time I came here, I waited in line for 20 minutes.  When it was my turn, I realized I left my wallet in the car.  It hurt so bad, I didn't come back for a year.\\n\\nI can walk to this place from my house- which is dangerous because those biscuits are just <font color = green>OH</font> <font color = green>SO</font> <font color = green>DREAMY</font>.  I can't describe them.  Just get some.\\n\\nDo I feel guilty about noshing on fabulous Strawberry <font color = green>Napoleons</font> and Jewish Pizza (kind of like a modified, yet TOTALLY delicious fruitcake bar) at 10:15am?  Hecks, naw... But they do have quiche and some other breakfast-y items for those who prefer a more traditional approach to your stomach's opening ceremony.\\n\\nJust go early :)  They open at 10 on Saturdays.  And bring cash...it's easier that way.</td>\n",
       "      <td>The first time I came here, I waited in line for 20 minutes.  When it was my turn, I realized I left my wallet in the car.  It hurt so bad, I didn't come back for a year.\\n\\nI can walk to this place from my house- which is dangerous because those biscuits are just <font color = red>OOOOH</font> <font color = red>EVEN</font> <font color = red>SULTRY</font>.  I can't describe them.  Just get some.\\n\\nDo I feel guilty about noshing on fabulous Strawberry <font color = red>Bolsheviks</font> and Jewish Pizza (kind of like a modified, yet TOTALLY delicious fruitcake bar) at 10:15am?  Hecks, naw... But they do have quiche and some other breakfast-y items for those who prefer a more traditional approach to your stomach's opening ceremony.\\n\\nJust go early :)  They open at 10 on Saturdays.  And bring cash...it's easier that way.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>We decided to give brunch a try for our first visit to Casbah.  We were surprised  by the huge tent covering the \\\"\"outdoor\\\"\" dining area.  We opted for an inside table, the interior is somewhat small the tables are close together.  For brunch, you are served your choice of drink, appetizer and entree.  \\n\\nFor our drinks, <font color = green>BJ</font> had a <font color = green>Bloody</font> <font color = green>Mary</font> and I had a Bellini.  We were served a basket of yummie bread and mini muffins.  For appetizers, we got a Three <font color = green>Sisters</font> <font color = green>Farms</font> mesclun greens and smoked salmon and truffled potato cake.  Very good.  For entrees we selected a jumbo lump crab & tomato omelet and the <font color = green>NY</font> strip steak.  Very relaxing and tasty meal.</td>\n",
       "      <td>We decided to give brunch a try for our first visit to Casbah.  We were surprised  by the huge tent covering the \\\"\"outdoor\\\"\" dining area.  We opted for an inside table, the interior is somewhat small the tables are close together.  For brunch, you are served your choice of drink, appetizer and entree.  \\n\\nFor our drinks, <font color = red>COCKSUCKING</font> had a <font color = red>Goddam</font> <font color = red>Newlyweds</font> and I had a Bellini.  We were served a basket of yummie bread and mini muffins.  For appetizers, we got a Three <font color = red>Brethren</font> <font color = red>Rural</font> mesclun greens and smoked salmon and truffled potato cake.  Very good.  For entrees we selected a jumbo lump crab & tomato omelet and the <font color = red>BROOKLYN</font> strip steak.  Very relaxing and tasty meal.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>And so....the search for a new hair salon continues. <font color = red>Sigh</font>. Don't get me wrong, the cut was a good cut. The salon itself was clean and stylish. The owner, welcoming and friendly. \\n\\nNow what went wrong. The cut was good, but it certainly wasn't what I expected from a salon with the reputation of Izzazu. I wasn't bowled over by my stylist's professionalism either. Don't diss my previous stylist....she rocked....you don't do yourself any favors by knocking someone else.  (And come on, I was WAAAYYYY overdue for a cut since I've been driving to Cleveland for a style.)  That being said, for $55 (and saving big bucks on gas, tolls, lunch and shopping) the cut was still a deal. But, when I started to sign the charge slip, it said $65, not $55. \\\"\"But,\\\"\" I said, \\\"\"the website said it was $55 for a Master stylist.\\\"\" \\\"\"Oh,\\\"\" the chick at the counter said, \\\"\"that's for Men's cuts.\\\"\" Silly me. \\n\\nSo when I got back to the office, I went online and checked. Nope, it said $55 for a Master Stylist WOMEN's haircut. Hmmmmm. So I called. The chick at the counter now said, \\\"\"Oh, our stylist's charge whatever they feel the cut SHOULD be.\\\"\" What?????? So I quoted the prices to her from the Izzazu website. She changed her tune again. \\\"\"Oh, well.....I'll refund you $10 if you give me your credit card number.\\\"\" Didn't she have my slip with the card number? \\\"\"Sorry, I don't give my credit card number over the phone.\\\"\" \\\"\"Or I can send you a gift certificate.\\\"\" \\\"\"Nope,\\\"\" I said through clenched teeth, \\\"\"I won't be coming back.\\\"\"\\n\\nIt wasn't the cut. It was the bait and switch. I'd gladly have paid it had they been up front and above-board ahead of time. As Judge Judy says, \\\"\"Don't pee on my leg and tell me it's raining.\\\"\" \\n\\nThe search goes on. Or I'll be back in Cleveland in the spring for the next cut!\\n\\nP. S.  One amusing side note:  I checked in at Izzazu when I arrived.  Turns out, I'm the Duchess!  The Duchess is displeased.</td>\n",
       "      <td>And so....the search for a new hair salon continues. <font color = green>Inhales</font>. Don't get me wrong, the cut was a good cut. The salon itself was clean and stylish. The owner, welcoming and friendly. \\n\\nNow what went wrong. The cut was good, but it certainly wasn't what I expected from a salon with the reputation of Izzazu. I wasn't bowled over by my stylist's professionalism either. Don't diss my previous stylist....she rocked....you don't do yourself any favors by knocking someone else.  (And come on, I was WAAAYYYY overdue for a cut since I've been driving to Cleveland for a style.)  That being said, for $55 (and saving big bucks on gas, tolls, lunch and shopping) the cut was still a deal. But, when I started to sign the charge slip, it said $65, not $55. \\\"\"But,\\\"\" I said, \\\"\"the website said it was $55 for a Master stylist.\\\"\" \\\"\"Oh,\\\"\" the chick at the counter said, \\\"\"that's for Men's cuts.\\\"\" Silly me. \\n\\nSo when I got back to the office, I went online and checked. Nope, it said $55 for a Master Stylist WOMEN's haircut. Hmmmmm. So I called. The chick at the counter now said, \\\"\"Oh, our stylist's charge whatever they feel the cut SHOULD be.\\\"\" What?????? So I quoted the prices to her from the Izzazu website. She changed her tune again. \\\"\"Oh, well.....I'll refund you $10 if you give me your credit card number.\\\"\" Didn't she have my slip with the card number? \\\"\"Sorry, I don't give my credit card number over the phone.\\\"\" \\\"\"Or I can send you a gift certificate.\\\"\" \\\"\"Nope,\\\"\" I said through clenched teeth, \\\"\"I won't be coming back.\\\"\"\\n\\nIt wasn't the cut. It was the bait and switch. I'd gladly have paid it had they been up front and above-board ahead of time. As Judge Judy says, \\\"\"Don't pee on my leg and tell me it's raining.\\\"\" \\n\\nThe search goes on. Or I'll be back in Cleveland in the spring for the next cut!\\n\\nP. S.  One amusing side note:  I checked in at Izzazu when I arrived.  Turns out, I'm the Duchess!  The Duchess is displeased.</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "pd.options.display.max_colwidth = 480 # increase column width so we can actually read the examples\n",
    "\n",
    "from IPython.core.display import display, HTML\n",
    "display(HTML(logger.df[['original_text', 'perturbed_text']].to_html(escape=False)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Conclusion\n",
    "\n",
    "Our constraint seems to have done its job: it filtered out attacks that did not swap out a named entity for another, according to the NLTK named entity detector. However, we can see some problems inherent in the detector: it often thinks the first word of a given sentence is a named entity, probably due to capitalization. \n",
    "\n",
    "We did manage to produce some nice adversarial examples! \"Sigh\" beacame \"Inahles\" and the prediction shifted from negative to positive."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
